Assigning Document Identifiers to Enhance Compressibility of Fulltext Indices

Index compression has been a major issue in the field of Information Retrieval Systems. In particular, due to the impressive figures involved with Web Search Engines (WSEs) the compression of the index is not an option anymore but it has become a must. The most important index compression methods are designed to work for Inverted File (IF) indexes. These methods are based on the assumption that the posting lists are stored as sequences of d gaps (i.e. differences among successive document identifiers). The compression is thus carried out by using variable length encoding methods which represents smaller number using a smaller number of bits. In this paper, instead of focusing on finding a novel encoding method, we propose an algorithm which allows the assignment of identifiers to documents in a way that minimizes the average values of d gaps. The simulations performed on a real dataset, i.e. the Google contest collection, show that our approach allows to obtain an IF index which is, depending on the d gap encoding chosen, up to 23% smaller than the one built over randomly assigned document identifiers. Moreover, we will show, both analytically and empirically, that the complexity of our algorithm is linear in space and time.